010-04-002 
Anderson 

Human-Computer Interface Incorporating Personal and Application 

Domains 

This invention was made with Government support under Contract DE-AC04- 
94AL85000 awarded by the U. S. Department of Energy. The Government has certain 
rights in the invention. 

Priority Claim 

This application claims the benefit of U.S. Provisional Application 60/202,448, filed on 
05/06/2000, incorporated herein by reference, and U.S. Patent Application 09/649,853, 
filed on 8/29/2000, incorporated herein by reference. 

Background of the Invention 

This invention relates to the field of human-computer interfaces, specifically those relating to 
haptics, multidimensional displays and navigation, interaction with multidimensional environments 
and objects, and methods of intuitively interfacing therewith. 

Computing technology has seen a many-fold increase in capability in recent years. Processors 
5 work at ever higher rates; memories are ever larger and always faster; mass storage is larger 
and cheaper every year. Computers now are essential elements in many aspects of life, and are 
often used to present three dimensional worlds to users, in everything from games to scientific 
visualization. 

The interface between the user and the computer has not seen the same rate of change. 
10 Screen windows, keyboard, monitor, and mouse are the standard, and have seen little change 
since their introduction. Many computers are purchased with great study as to processor speed, 
memory size, and disk space. Often, little thought is given to the human-computer interface, 
although most of the user's experience with the computer will be dominated by the interface 
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(rarely does a user spend significant time waiting for a computer to calculate, while every 
interaction must use the human-computer interface). 

As computers continue to increase in capability, the human-computer interface will become 
increasingly important. The effective bandwidth of communication with the user will not be 
5 sufficient using only the traditional mouse and keyboard for input and monitor and speakers for 
output. More capable interface support will be desired to accommodate more complex and 
demanding applications. For example, six degree of freedom input devices, force and tactile 
feedback devices, three dimensional sound, and stereo or holographic displays can improve the 
human-computer interface. 

10 As these new interface capabilities become available, new interface methods are needed to 
fully utilize the new modes of human-computer communication enabled. Preferably, such new 
methods can build on interface methods learned by users in real-world situations and in two- 
dimensional computer interfaces. Specifically, users have become accustomed to two- 
dimensional computer interfaces, with control panels, menus, and buttons. Users are also 

15 accustomed to three-dimensional interfaces in real-world situations, visual and tactile depth 

perception aid in finding and manipulating such controls. For example, the controls of a car radio 
are always in some understandable relationship to the usual position of the driver. Such simplistic 
relationship can be undesirable in a three-dimensional computer environment, however, where 
the user can desire that controls be available during three-dimensional navigation free of the 

20 requirement of experiencing the environment through a windshield. 

Fuller realization of the entertainment and productivity benefits possible from computer 
technology requires more improved interfaces. Accordingly, there is a need for improved human- 
computer interfaces that offer more intuitive navigation and control of the computer. 

Summary of the Invention 

25 The present invention provides a human-computer interface. The interface includes provision 
of an application domain, for example corresponding to a three-dimensional application. The user 
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is allowed to navigate and interact with the application domain. The interface also includes a 
personal domain, offering the user controls and interaction distinct from the application domain. 
The separation into two domains allows the most suitable interface methods in each: for 
example, three-dimensional navigation in the application domain, and two- or three-dimensional 
5 controls in the personal domain. Transitions between the application domain and the personal 
domain are under control of the user, and the transition method is substantially independent of 
the navigation in the application domain. For example, the user can fly through a three- 
dimensional application domain, and always move to the personal domain by moving a cursor 
near one extreme of the display. 

10 The arrangement of domains, the method of controlling transitions between domains, and the 
method of communicating when a transition occurs can all have significant effect on the overall 
effectiveness of the human-computer interface. Accordingly, the present invention comprises 
various arrangements of personal domains and application domains, corresponding to various 
desirable human-computer interface characteristics. The present invention further comprises 

15 various techniques for allowing a user to control transition between domains, and for 
communicating to the user when a transition occurs. 

Advantages and novel features will become apparent to those skilled in the art upon 
examination of the following description or may be learned by practice of the invention. The 
objects and advantages of the invention may be realized and attained by means of the 

20 instrumentalities and combinations particularly pointed out in the appended claims. 

Description of the Figures 

The accompanying drawings, which are incorporated into and form part of the specification, 
illustrate embodiments of the invention and, together with the description, serve to explain the 
principles of the invention. 
25 Figure 1 is an illustration of a computer display representing an application domain and a 
personal domain. 
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Figure 2(a,b) is an illustration of a computer display representing an application domain and a 
personal domain, showing the display when each domain is the active domain. 
Figure 3(a,b,c) is an illustration of a computer display representing an application domain and a 
personal domain, showing the display when each domain is the active domain and showing a 
5 transition from the application domain as the active domain to the personal domain as the active 
domain. 

Figure 4(a,b,c,d) is an illustration of a computer display representing an application domain and a 
plurality of personal domains, showing the display when each domain is the active domain. 
Figure 5 is an illustration of the correspondence of one personal domain with three application 
10 domains. 

Figure 6 is an illustration of a three-dimensional application domain and three personal domains. 
Figure 7 is a flow diagram of an example implementation of the present invention. 
Figure 8 is a schematic diagram of an example implementation of the present invention. 
Figure 9 is an illustration of an application domain and a personal domain, as desired to be 
1 5 presented to a user. 

Figure 10 is an illustration of the range of motion of the input device in the two domains. 

Detailed Description of the Invention 

The present invention provides a human-computer interface. The interface includes provision 
of two distinct domains, and detection of transitions between the two. The interface provides an 

20 application domain, for example corresponding to a three-dimensional application. The user is 
allowed to navigate and interact with the application when the application domain is active. The 
interface also provides a personal domain, offering the user controls and interaction distinct from 
the application domain. The separation into two domains allows the most suitable interface 
methods in each: for example, navigation in the application domain, and controls in the personal 

25 domain. Transitions between the application domain and the personal domain are under control 
of the user, and the transition method can be substantially independent of the navigation in the 



010-04-002; ANDERSON; Page 4 



application domain. For example, the user can fly through a three-dimensional application 
domain, and ajways move to the personal domain by moving a cursor near one extreme of the 
display or one extreme of the application domain. 

The arrangement of domains, the method of controlling transitions between domains, and the 
5 method of communicating when a transition occurs can all have significant effect on the overall 
effectiveness of the human-computer interface. Accordingly, the present invention comprises 
various arrangements of personal domains and application domains, corresponding to various 
desirable human-computer interface characteristics. The present invention further comprises 
various techniques for allowing a user to control transition between domains, and for 
10 communicating to the user when a transition occurs. 

The Application Domain 

The application domain corresponds to an application, and provides interaction with the user 
according to the interface characteristics appropriate to the application. For example, if the 
application includes a space navigable in three dimensions, the application domain can provide 

15 display of the three-dimensional space, and interaction comprising navigation in three 

dimensions. Interaction in the application domain includes detection of user input to designate 
the personal domain as the active domain. Active domain transitions are discussed below. 
Designation of the personal domain as the active domain can be independent of control of the 
application so that the user always has an intuitive path back to the personal domain. Reliance 

20 on ready and consistent access to the personal domain can improve the effectiveness of the 

human-computer interface. Designation of the personal domain as the active domain can be in a 
consistent manner across multiple applications, allowing a user to rely on the availability and 
interface characteristics if the personal domain instead of learning new interface characteristics 
for each application. Further, the interaction in the application domain can be consistent across 

25 multiple applications, allowing a user to work with multiple applications without requiring 
adaptation to different application interfaces. 
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The Personal Domain 

The personal domain corresponds to a personal interaction environment, and can include 
interaction with characteristics of the application and interaction external to the application. For 
example, the personal domain can include controls over aspects of the application, including 

5 navigation in a three-dimensional application space. It can also include interaction such as 

communications (e.g., mail, inter-computer communications) and data operations (e.g., printing, 
file organization and operations). The interaction environment of the personal domain can be 
distinct from the application domain, since the personal domain can focus on personal interaction 
needs rather than interaction required by the application. The interaction environment of the 

10 personal domain can also be substantially consistent across many application domains, allowing 
the user to learn a single interface paradigm useful in many applications. 

The personal domain includes detection of the user designating the application domain as the 
active domain. Transitions from the personal domain to the application domain can be different 
from transitions from the application domain to the personal domain, without confusion to the 

15 user since the interaction environments are distinct. The personal domain can also include 
detection of transitions to other personal domains, and to multiple application domains. A user 
can transition from an application domain to a personal domain. While in the personal domain, 
the user can interact with the computer according to a familiar personal interaction environment. 
The user can transition from the personal domain back to the application domain, or to other 

20 personal domains tailored to specific interaction needs (e.g., interactions related to computer 
communications can be grouped in a single personal domain). The user can also transition from 
the personal domain to other application domains. 

Transitions Between Domains 

Provision of natural, intuitive transitions between active domains can be important to the 
25 overall effectiveness of the human-computer interface. Each transition preferably is accessible to 
the user via intuitive controls, for example with computer metaphors of physical world 
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interactions. Each transition can also include feedback to the user so that the user is always 
certain which domain is active. 

Control of a cursor in multidimensional spaces can provide intuitive transition controls. An 
active domain can include a boundary between it and another domain. The user can effect a 
5 transition to the other domain by causing the cursor to traverse the boundary. Once in the new 
domain, the boundary might be reset to correspond to the interaction environment of the new 
domain. The boundary can be visible, such as a line or region shown on the display. For 
example, the application domain can use the majority of a display for application purposes, and 
reserve a portion at the bottom of the display for a boundary to a personal domain. Moving the 

10 cursor into that lower portion causes a transition to the personal domain. The personal domain 
can now use the entire display for personal domain interaction, and can use a different portion of 
the display, or a different transition control, to indicate transition back to the application domain. 
Further, the correspondence of input device with computer interaction can be different in different 
domains. For example, a three-dimensional interface device can be active in the application 

15 domain and a two dimensional device in the personal domain. As another example, the cursor 
motion, speed, and resulting navigation can be in different relation to the input device in different 
domains. 

The boundary can also be invisible, such as by corresponding to a consistent portion of the 
domain or a consistent motion of a cursor control input device. For example, moving a cursor off 

20 the bottom edge of a display in the application domain can cause a transition to a personal 

domain. As another example, if the cursor is controllable in three dimensions, moving the cursor 
toward the user can cause a transition to a personal domain (i.e., the user "pulls out" of the 
application domain). The transition can also be relative to the motion of an input device. For 
example, moving a cursor control device to an extreme of its range of motion can cause a 

25 transition between domains (e.g., the user can get to the personal domain by moving the input 
device all the way back, or by making a specific set of motions with the input device). 
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The occurrence of a transition between domains can be communicated to the user. The 
characteristics of the display presented to the user can be distinct among the different domains. 
The display of the new domain can displace that of the former domain in a perceptible manner, 
for example by sliding in front of the former display, pushing the former display off, or rotating the 
5 former display out of view. The imminence of a transition can also be communicated to the user 
by providing force feedback to a user input device. The force feedback, for example, can be a 
click (as though the user dragged a pen across a crack in a surface) or a bump (as though the 
user was moving between physical spaces with a raised boundary). 

The imminence of a transition can also be communicated to the user, and unplanned 
10 transitions avoided, by applying a force resisting the motion that would cause a transition. For 
example, if boundary traversal causes a transition, a force substantially opposing motion toward 
the boundary can be applied to a user input device. The occurrence of such a transition can be 
communicated by reducing the resistive force when the domain transition occurs, providing the 
user with haptic feedback of a sensation of popping through a barrier between domains. 

15 Example of Interfacing with a personal domain and an application domain 

Figure 1 is an illustration of a computer display representing an application domain and a 
personal domain. The application domain is represented to the user as an application portion 101 
of the display 100. The personal domain is represented to the user as a personal portion 102 of 
the display 100. The user controls which domain is currently active, and interfaces with the 

20 computer according to the interface characteristics of the currently active domain. For example, 
the user can control the position of a cursor, and designate the active domain by moving the 
cursor to the portion of the display corresponding to the domain desired to be the active domain. 

Moving the cursor to a defined region of the application portion of the display (in the figure 
moving the cursor to the bottom of the application portion) can switch the active domain to be the 

25 personal domain. While in the personal domain, the user can be presented with an interface 
having characteristics distinct from those of the application domain. For example, the personal 
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domain can offer interface characteristics applicable to a wide variety of applications, providing 
the user a consistent and intuitive way to access a consistent set of system interface tools. The 
personal domain can offer interface tools affecting the application domain, and can offer interface 
tools affecting other than the application domain. For example, the personal domain can offer 
5 tools affecting the scale or content of the application domain, and can also offer tools allowing 
communication with other applications, other users, or other features of the computer. 

Moving the cursor to a defined region of the personal portion (the top in the figure) can switch 
the active domain to be the application domain. The relationship between the application domain 
and the personal domain can be substantially independent of interaction in the application 

10 domain, allowing the user a consistent and intuitive way to access the personal domain. For 
example, the user can navigate a three-dimensional application domain with few restrictions, 
flying, teleporting, zooming in or out, moving objects, etc. At all times in the application domain, 
the controls of the personal domain are accessible in the same manner, providing easy access to 
the interface characteristics of the personal domain. 

1 5 The user can control transitions by movement of a cursor (visible or invisible). An input device 
can also have its range of motion mapped such that a transition between domains is indicated by 
moving the device to a defined region of the range of motion. For example, a joystick as an input 
device can have the extreme back portion of its range of motion correspond to a transition to the 
personal domain, and the extreme forward portion of its range of motion correspond to a 

20 transition to the application domain. The transition definition can be dependent on which domain 
is currently active. 

The interface can also provide feedback to allow the user to manage transitions across the 
boundary between the application domain and the personal domain. A force fed back to an 
appropriate input device, directed substantially away from the boundary, can provide the user a 
25 haptic indication of the boundary. The force can decrease perceptibly as the transition occurs, 
providing the user a sensation of pushing through a boundary and entering the newly active 
domain. The haptic feedback can allow effective designation of the active domain without 
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requiring careful visual management of the cursor by the user, and can help prevent inadvertent 
active domain transitions. 



Example of changing display as the active domain changes 

Figure 2(a,b) is an illustration of a computer display. In figure 2a the application domain is the 

5 active domain. Most of the display 200 is devoted to the application portion 201 ; the user can be 
effectively immersed in the application. The user can still, however, access the personal domain 
in an intuitive and consistent manner. The personal domain is represented in Figure 2a as a 
small portion 202 of the display; it can be invisible as long as the user can reach it in an intuitive 
manner. The user can switch the active domain to be the personal domain by moving the cursor 

10 toward the bottom of the application portion of the display, whether into the small portion as 
shown in Figure 2a or to the bottom if the personal portion is not visible. Figure 2b shows the 
display changed as the personal domain becomes the active domain. The application portion 202 
consumes a larger portion of the display 200. The user can interact with the personal domain and 
make use of the larger portion of the display now available for the personal domain. 

15 Example of three-dimensional display change as the active domain changes 

Figure 3(a,b,c) is an illustration of a computer display with the active domain the application 
domain (Figure 3a), the personal domain (Figure 3c), and a transition between the 
two (Figure 3b). In Figure 3a, the application domain is the active domain. The entire display 300 
can be used for the application, and the user can be immersed in the interface characteristics of 

20 the application. Moving the cursor to a defined region of the display, the right edge in the figure, 
indicates a transition to the personal domain as the active domain. Figure 3b shows the display 
change as the interface accommodates the transition from the application domain to the personal 
domain. The display reflects an animation of the application domain moving off the display and 
the personal domain moving into its place. The figure shows the transition as an illusion of the 

25 user turning the head from looking at the application domain to the side where the personal 
domain resides. Making the transition from the application domain to the personal domain 
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substantially independent of navigation and control in the application domain provide the user an 
interface that allows interaction with the application while keeping the interface of the personal 
domain always accessible just by "looking" to the side (moving the cursor to the right). The user 
can move back to the application domain by simply moving back to the left of the personal 
5 domain display. 

Example of multiple personal domains, with three-dimensional interface experience 
Figure 4(a,b,c,d) is an illustration of a computer display associated with an application domain 
and three personal domains. The application domain occupies the majority of the display 400 
when the application domain is the active domain. The interface provides the user with three 

10 personal domains, one below the application domain and one at either side of the application 
domain (shown virtually in the figure extending from the display). The personal domains at the 
sides can be accessible via the sides of the application domain, the sides of the lower personal 
domain, or both, depending on the interface experience desired. The user can interact with the 
application domain while it is active, and always have access to interface tools in the three 

15 personal domains by merely causing transition to one of them as active. For example, the user 
can interact with objects in a three-dimensional sculpting application. Moving the cursor to the left 
can provide the user access to that personal domain, perhaps providing access to interface tools 
that customize the characteristics of the sculpting application. Moving the cursor to the right 
when in that personal domain moves the user back to the application domain. Moving the cursor 

20 to the bottom of the application domain causes that personal domain to become active, perhaps 
providing access to other applications or other users. Moving the cursor back up returns the user 
to the application domain. Moving the cursor to the right provides access to that personal 
domain, perhaps providing access to computer system tools (e.g., printers and file systems). 

Example of personal domain independent of application domains 
25 The interface can provide the user with personal domain interface characteristics consistent 
across diverse applications. Each application can have its own interface characteristics, as 
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dictated by the needs of the application. Diverse applications, however, can provide similar 
methods for transitioning to the personal domain. Once in the personal domain, the user can be 
presented with familiar interface characteristics. 

Figure 5 is an illustration of a single personal domain offering transitions to multiple (three in 
5 the figure) application domains. Each application 501, 502, 503 transitions to the personal 
domain 504 by moving the cursor to the right edge of the display. Once in the personal 
domain 504, the user can interact according to the personal domain interface characteristics. The 
personal domain provides transitions to three different application domains 501, 502, 503; the 
user transitions to one by moving the cursor to that region of the personal domain. 

10 Example of personal domains in connection with a three-dimensional navigable space 
Figure 6 is an illustration of three personal domains 602, 603, 604 and an application 
domain 601. Application domain 601 corresponds to an application space navigable in three 
dimensions: app-x, app-y, and app-z. When the application domain is the active domain, the user 
can navigate therein, using almost the full range of the three dimensional space or three 

15 dimensional input device. Moving a cursor to a portion of the application space nearest the user 
(or smallest range of app-z), for example by moving the cursor in the app-z direction or by 
moving the input device toward the corresponding extreme of its range of motion, can cause a 
transition to the first personal domain 602 as the active domain. A resistive force can be applied 
as the cursor nears the transition region to provide more positive user control of the transition, 

20 and then reduced when the transition occurs to provide more positive communication of the 
transition. 

When the first personal domain 602 is the active domain, the user can be presented with an 
interaction environment corresponding to the first personal domain. The interaction environment 
can comprise two dimensions, corresponding to just the plane of the boundary in the figure. 
25 Alternatively, the interaction environment can comprise three dimensions, for example 
comprising three dimensional buttons or controls and three-dimensional haptic feedback. 
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Transitions to other domains can be similar to as discussed for the application to first personal 
domain transition, except that the first personal domain provides access to two other personal 
domains as well as back to the application domain. Moving to the far portion of the first personal 
domain can cause a transition back to the application domain, optionally including force feedback 
5 as discussed before. Moving to the far left of the first personal domain can cause a transition to 
the second personal domain. Moving to the far right of the first personal domain can cause a 
transition to the third personal domain. Once in the second or third personal domains, the user 
can again use almost the full range of the three dimensional space for interaction in the domain 
(similar to the discussion for the first personal space). Moving back across the corresponding 

10 edge can cause a transition back to the first personal domain. 

Other mappings of domains and domain boundaries can be used, including boundaries 
beyond those possible with conventional three dimensional spaces. For example, the second and 
third personal domains in Figure 6 can be directly linked via a domain boundary at their lower 
edges, not possible with a physical three dimensions. The mapping shown in Figure 6 does, 

15 however, present the user with an intuitive, physical world sensation: the personal domains are 
intuitively in front of the application domain, and once in the first personal domain the second and 
third domains are at the user's sides. Force feedback can enhance the physical world sensation: 
the user can interact in the application domain, then pop through the front wall into the personal 
domain, interact there, then pop back into the application domain or the other personal domains. 

20 Example Implementation 

A human-computer interface incorporating the present invention can be implemented with 
computer programming tools and techniques known to those skilled in the art. As an example, 
Figure 7 is a flow diagram of one suitable implementation, accommodating multiple domains, with 
cursor/boundary transitions between domains, and with force feedback near the boundaries. 

25 The interface comprises display and interaction according to the active domain 701 . As 

discussed above, the display and interaction can be distinct between domains. The user input is 



010-04-002; ANDERSON; Page 13 



monitored to detect when the cursor exceeds a boundary between the present active domain and 
another domain 702. If it has, then any force feedback relevant to the boundary is reduced 703, 
the active domain changed to the new domain 704, and the display and interaction replaced with 
those corresponding to the newly active domain 701 . If it has not, then a determination is made 
5 whether the cursor is nearing an inter-domain boundary 705. If it is, then an appropriate feedback 
force is applied 706. Since the cursor is not past the boundary, the display and interaction are still 
according to the presently active domain 701 . 

Another illustration of an implementation of the present invention is shown in Figure 8. An 
application domain function ADF effects a display and interaction according to the application 

10 domain. An active domain controller ADC monitors for transitions indicated by the user to other 
domains, for example by cursor motion. When the ADC detects such a transition, control is 
transferred to the other domain, in the figure to personal domain function PDF. The PDF 
communicates the transition to the user, for example by reducing force feedback and by moving 
the display of the personal domain in place of the display of the application domain. The PDF 

15 then provides display and interface according to the personal domain. The ADC still monitors for 
transitions indicated by the user to other domains, for example by cursor motion. When the ADC 
detects such a transition, control is transferred to the function implementing the other domain. 
The figure shows only two domains; the number of domains and mapping between domains can 
be as needed to achieve the desired user interface characteristics. 

20 Example Domain and Input Device Mapping 

Figure 9 is an illustration of an application domain and a personal domain, as desired to be 
presented to a user. The user can interact with the domains using an input device moveable in 
three dimensions: x-device, y-device, and z-device. The input device can have limited range of 
motion, however. For efficient interaction with either domain, it can be desirable that a majority of 

25 the range of motion of the input device be dedicated to the active domain. 
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The range of motion of the input device can be mapped based on the active domain. As an 
example, Figure 10 is an illustration of the range of motion of the input device in the two 
domains. On the left, the rear portion of the range of motion (x-device, y-device, and z-device 
space) is mapped to interactions with the application domain. A smaller front portion of the space 
5 is mapped to a personal transition portion. The user thus has most of the input device's range of 
motion available for interaction with the application domain. The user can indicate a transition to 
a personal domain by moving the input device into the personal transition portion of the space. 
On the right is an illustration of the mapping of the space after transition to a personal domain. 
The front portion of the space, mapped to interaction with the personal space, is larger than on 

10 the left, allowing more of the input device range of motion to be used in personal space 
interactions. A smaller rear portion of the space is mapped to indicate transitions to an 
application domain. The figure shows a rectangular space with a planar boundary; the space and 
boundaries can have various shapes depending on the input device characteristics and the 
specific user interaction experience desired. 

15 The display can also be configured to enhance the interaction according to each domain. As 
an example, objects in the personal domain can be displayed at one size when the user is 
interacting according to the application domain, and at another, larger size when the user is 
interacting with the personal domain. As another example, objects in the personal domain can be 
displayed with differing visual intensities depending on which domain is active. As another 

20 example, objects in the personal domain can be displayed as transparent or semitransparent 

when the user is interacting with the application domain, and as solid when the user is interacting 
with the personal domain. 

The particular sizes and equipment discussed above are cited merely to illustrate particular 
embodiments of the invention. It is contemplated that the use of the invention may involve 

25 components having different sizes and characteristics. It is intended that the scope of the 
invention be defined by the claims appended hereto. 
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